Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Improve ASA/FTD Ingest Pipeline #23766

Merged

Conversation

hitchatwork
Copy link
Contributor

@hitchatwork hitchatwork commented Jan 29, 2021

What does this PR do?

Fixes #21658

For messages 716002:

  • Changed to GROK; allows for better parsing of event.reason
  • Added field for cisco.webvpn.group_name
  • Added field for event.reason per cisco docs for why session was terminated
  • Added field for cisco.termination_user for the AAA username terminating the connection

For messages 722051:

  • Add angle brackets to dissect to properly dissect the message, per cisco docs
  • Added field for cisco.webvpn.group.name

For messages 305011:

  • Change to GROK; allows for variance in message format with identity firewall

For messages 302020:

  • Added GROK pattern to allows for variance in message format with identity firewall

For messages 302014/302016/302021:

  • Added patterns and modified order of patterns of GROK to better match teardown messages
  • Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match.
  • Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future.
  • Added icmp_type and icmp_code parsing for 302021 messages
  • Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME.

And:

  • Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor.
  • Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.
  • Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.

Why is it important?

Improved parsing of ASA/FTD messages mentioned above.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

cd x-pack/filebeat
PYTEST_ADDOPTS="-k cisco" TESTING_FILEBEAT_FILESETS=asa,ftd mage -v pythonIntegTest

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 29, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jan 29, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #23766 updated

  • Start Time: 2021-03-24T15:18:18.927+0000

  • Duration: 125 min 49 sec

  • Commit: 0c52b2d

Test stats 🧪

Test Results
Failed 0
Passed 13173
Skipped 2243
Total 15416

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 13173
Skipped 2243
Total 15416

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 29, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

Copy link
Member

@P1llus P1llus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding some small comments

@jsoriano
Copy link
Member

jsoriano commented Feb 1, 2021

jenkins run the tests please

@hitchatwork hitchatwork requested review from a team as code owners February 12, 2021 19:55
@botelastic botelastic bot added Team:Automation Label for the Observability productivity team Team:Ingest Management labels Feb 12, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ingest-management (Team:Ingest Management)

@hitchatwork hitchatwork force-pushed the hitchatwork-ImproveASAPipeline-1 branch from 6bfca13 to ae59310 Compare February 13, 2021 00:27
@hitchatwork
Copy link
Contributor Author

@jsoriano - could you kick jenkins off again? I made adjustments from @P1llus comments and also updated from master. Thanks!

@andrewkroh
Copy link
Member

jenkins, run tests

@hitchatwork
Copy link
Contributor Author

Thanks for kicking that off @andrewkroh - 4 of the tests failed (dictionary item adds due to ICMP processing). I decided to simplify the PR by removing any changes to ICMP at all (will create a seperate PR for that). Could you kick Jenkins off again?

@andrewkroh
Copy link
Member

jenkins, run tests

@hitchatwork
Copy link
Contributor Author

I'm not sure why this build failed. Looks like something maybe in the backend with the Windows 2019 environment setup for the build?

@andrewkroh
Copy link
Member

jenkins, run tests

@hitchatwork
Copy link
Contributor Author

Thanks a lot @andrewkroh - looks like it passed and can be reviewed by the relevant teams. Appreciate the assistance.

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work on this. It LGTM, but it needs an entry in the changelog to describe the change. Can you please put a note into CHANGELOG.next.asciidoc at the root of the repo.

@hitchatwork
Copy link
Contributor Author

Thanks for your work on this. It LGTM, but it needs an entry in the changelog to describe the change. Can you please put a note into CHANGELOG.next.asciidoc at the root of the repo.

Added changelog entry.

@andrewkroh
Copy link
Member

Looks like we have a conflict in fields.go due to the fields changes. This will need a rebase.

@hitchatwork hitchatwork force-pushed the hitchatwork-ImproveASAPipeline-1 branch 2 times, most recently from 0a80440 to 5bbd0c5 Compare February 22, 2021 00:56
@hitchatwork
Copy link
Contributor Author

Looks like we have a conflict in fields.go due to the fields changes. This will need a rebase.

@andrewkroh - I have rebased and cleaned up some of the commits. LMK if something else needs work. Thanks!

@andrewkroh
Copy link
Member

jenkins, run tests

@hitchatwork
Copy link
Contributor Author

@andrewkroh if you could take a look, seems it's stuck in build or something odd. Thanks!

@andrewkroh
Copy link
Member

jenkins, run tests

@hitchatwork
Copy link
Contributor Author

@andrewkroh looks like the build is failing from when you pushed in the golden files commit. Looks like expected output for cisco AMP module changed, and that's failing the build.

@andrewkroh
Copy link
Member

jenkins, run tests

@andrewkroh andrewkroh force-pushed the hitchatwork-ImproveASAPipeline-1 branch 4 times, most recently from c090416 to 2d7997b Compare March 24, 2021 14:57
Fixes elastic#21658

For messages 716002:
- Changed to GROK; allows for better parsing of event.reason
- Added field for cisco.webvpn.group_name
- Added field for event.reason per cisco docs for why session was terminated
- Added field for cisco.termination_user for the AAA username terminating the connection

For messages 722051:
- Add angle brackets to dissect to properly dissect the message, per cisco docs
- Added field for cisco.webvpn.group.name

For messages 305011:
- Change to GROK; allows for variance in message format with identity firewall

For messages 302020:
- Added GROK pattern to allows for variance in message format with identity firewall

For messages 302014/302016/302021:
- Added patterns and modified order of patterns of GROK to better match teardown messages
- Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match.
- Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future.
- Added icmp_type and icmp_code parsing for 302021 messages
- Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME.

And:

- Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor.
- Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.
- Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.
@andrewkroh andrewkroh force-pushed the hitchatwork-ImproveASAPipeline-1 branch from 2d7997b to 0c52b2d Compare March 24, 2021 15:17
@andrewkroh andrewkroh merged commit b5e43fc into elastic:master Mar 24, 2021
@andrewkroh andrewkroh added needs_backport PR is waiting to be backported to other branches. and removed Team:Automation Label for the Observability productivity team Team:Ingest Management labels Mar 24, 2021
@andrewkroh andrewkroh added v7.13.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 24, 2021
andrewkroh pushed a commit to andrewkroh/beats that referenced this pull request Mar 25, 2021
Fixes elastic#21658

For messages 716002:
- Changed to GROK; allows for better parsing of event.reason
- Added field for cisco.webvpn.group_name
- Added field for event.reason per cisco docs for why session was terminated
- Added field for cisco.termination_user for the AAA username terminating the connection

For messages 722051:
- Add angle brackets to dissect to properly dissect the message, per cisco docs
- Added field for cisco.webvpn.group.name

For messages 305011:
- Change to GROK; allows for variance in message format with identity firewall

For messages 302020:
- Added GROK pattern to allows for variance in message format with identity firewall

For messages 302014/302016/302021:
- Added patterns and modified order of patterns of GROK to better match teardown messages
- Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match.
- Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future.
- Added icmp_type and icmp_code parsing for 302021 messages
- Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME.

And:

- Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor.
- Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.
- Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.

(cherry picked from commit b5e43fc)
andrewkroh added a commit that referenced this pull request Mar 25, 2021
Fixes #21658

For messages 716002:
- Changed to GROK; allows for better parsing of event.reason
- Added field for cisco.webvpn.group_name
- Added field for event.reason per cisco docs for why session was terminated
- Added field for cisco.termination_user for the AAA username terminating the connection

For messages 722051:
- Add angle brackets to dissect to properly dissect the message, per cisco docs
- Added field for cisco.webvpn.group.name

For messages 305011:
- Change to GROK; allows for variance in message format with identity firewall

For messages 302020:
- Added GROK pattern to allows for variance in message format with identity firewall

For messages 302014/302016/302021:
- Added patterns and modified order of patterns of GROK to better match teardown messages
- Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match.
- Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future.
- Added icmp_type and icmp_code parsing for 302021 messages
- Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME.

And:

- Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor.
- Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.
- Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.

(cherry picked from commit b5e43fc)

Co-authored-by: hitchatwork <56573831+hitchatwork@users.noreply.github.com>
@hitchatwork hitchatwork deleted the hitchatwork-ImproveASAPipeline-1 branch March 29, 2021 22:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Filebeat Cisco module doesn't correctly parse 305011, 302015, 302013, or 722051 message types
7 participants